Search Results: "wouter"

7 November 2015

Wouter Verhelst: Minidebconf Cambridge

My main reason for being here was to join the Debconf video team sprint. But hey, since I'm here anyway, why not join all of it, right? Right. Apart from the video team stuff that I've been involved in, I've been spending the last few days improving NBD: There're a few more things that I'd like to see fixed, but after that I'll probably release 3.12 (upstream and in Debian) sometime later this week. With that, I'll be ready to start tackling Debian bug #796633, to provide proper systemd support in nbd-client.

27 October 2015

Wouter Verhelst: Adventures with modern electronics

I got me a new television last wednesday. The previous one was still functional, but with its 20 inch display it was (by today's standards) a very small one. In itself that wasn't an issue, but combined with my rather large movie collection and the 5.1 surround set that I bought a few years ago, my living room was starting to look slightly ridiculous. The new TV is a Panasonic 40" UHD ("4K") 3D-capable so-called smart thing. Unfortunately, all did not go well. When I initially hooked it up, I found it confusingly difficult to figure out how to make it display something useful from the HDMI input rather than from the (not yet connected) broadcast cable. This eventually turned out to be due to the fact that the HDMI input was not selectable by a button marked "AUX" or similar, but by a button marked with some pictogram on a location near the teletext controls, which was pretty much the last place I'd look for such a thing. After crossing that bridge, I popped in the correct film for that particular day and started watching it. The first thing I noticed, however, was that something was odd with the audio. It turned out that the TV as well as the 5.1 amplifier support the CEC protocol, which allows the TV to control some functionality of the A/V receiver. Unfortunately, the defaults were set in the TV to route audio to the TV's speakers, rather than to the 5.1 amp. This was obviously wrong, and it took me well over an hour to figure out why that was happening, and how I could fix it. My first solution at that time was to disable CEC on the amplifier, so that I could override where the audio would go there. Unfortunately, that caused the audio and video to go out of sync; not very pleasant. In addition, the audio would drop out every twenty seconds or so, which if you're trying to watch a movie is horribly annoying, and eventually I popped the DVD into a machine with analog 5.1 audio and component HD video outputs; not the best quality, but at least I could stop getting annoyed about it. Over the next few days, I managed to get the setup working better and better: There's just on thing remaining: when I go into the channel list and try to move some channels around, the TV has the audacity to tell me that it's "not allowed". I mean, I paid for it, so I get to say what's allowed, not you, thankyouverymuch. Anyway, I'm sure I'll figure that out eventually. The TV also has some 3D capability, but unfortunately it's one of those that require active 3D glasses, so the set that I bought at the movie theatre a while ago won't work. So after spending several tens of euros on extra cabling, I'll have to spend even more on a set of 3D glasses. They'll probably be brand-specific, too. Ah well. It's a bit odd, in my opinion, that it takes me almost a week to get all that stuff to properly work. Ten years ago, the old TV had some connections, a remote, and that was it; you hook it up and you're done. Not anymore.

15 October 2015

Wouter Verhelst: On Planet Debian and the Code of Conduct

I am not going to talk about Norbert Preining's continuous ranting against Debian's Code of Conduct. Suffice to say that it annoys me, and that I think he is wrong. I am, however going to mention that contrary to his statements (penultimate paragraph), yes, the Code of Conduct is supposed to apply to Planet Debian as well. Its introductory paragraph states:
The Debian Project, the producers of the Debian system, have adopted a code of conduct for participants to its mailinglists, IRC channels and other modes of communication within the project.
Note the "other modes of communication within the project" bit. Planet Debian is planet.debian.org. If your blog is aggregated there, you are communicating within the project. At least that's my interpretation of it.

14 August 2015

Wouter Verhelst: Multi-pass transcoding to WebM with normalisation

Transcoding video from one format to another seems to be a bit of a black art. There are many tools that allow doing this kind of stuff, but one issue that most seem to have is that they're not very well documented. I ran against this a few years ago, when I was first doing video work for FOSDEM and did not yet have proper tools to do the review and transcoding workflow. At the time, I just used mplayer to look at the .dv files, and wrote a text file with a simple structure to remember exactly what to do with it. That file was then fed to a perl script which wrote out a shell script that would use the avconv command to combine and extract the "interesting" data from the source DV files into a single DV file per talk, and which would then call a shell script which used gst-launch and sox to do a multi-pass transcode of those intermediate DV files into a WebM file. While all that worked properly, it was a rather ugly hack, never cleaned up, and therefore I never really documented it properly either. Recently, however, someone asked me to do so anyway, so here goes. Before you want to complain about how this ate the videos of your firstborn child, however, note the above. The perl script spent a somewhat large amount of code reading out the text file and parsing it into an array of hashes. I'm not going to reproduce that, since the actual format of the file isn't all that important anyway. However, here's the interesting bits:
foreach my $pfile(keys %parts)  
        my @files = @ $parts $pfile ;
        say "#" x (length($pfile) + 4);
        say "# " . $pfile . " #";
        say "#" x (length($pfile) + 4);
        foreach my $file(@files)  
                my $start = "";
                my $stop = "";
                if(defined($file-> start ))  
                        $start = "-ss " . $file-> start ;
                 
                if(defined($file-> stop ))  
                        $stop = "-t " . $file-> stop ;
                 
                if(defined($file-> start ) && defined($file-> stop ))  
                        my @itime = split /:/, $file-> start ;
                        my @otime = split /:/, $file-> stop ;
                        $otime[0]-=$itime[0];
                        $otime[1]-=$itime[1];
                        if($otime[1]<0)  
                                $otime[0]-=1;
                                $otime[1]+=60;
                         
                        $otime[2]-=$itime[2];
                        if($otime[2]<0)  
                                $otime[1]-=1;
                                $otime[2]+=60;
                         
                        $stop = "-t " . $otime[0] . ":" . $otime[1] .  ":" . $otime[2];
                 
                if(defined($file-> start )   defined($file-> stop ))  
                        say "ln " . $file-> name  . ".dv part-pre.dv";
                        say "avconv -i part-pre.dv $start $stop -y -acodec copy -vcodec copy part.dv";
                        say "rm -f part-pre.dv";
                  else  
                        say "ln " . $file-> name  . ".dv part.dv";
                 
                say "cat part.dv >> /tmp/" . $pfile . ".dv";
                say "rm -f part.dv";
         
        say "dv2webm /tmp/" . $pfile . ".dv";
        say "rm -f /tmp/" . $pfile . ".dv";
        say "scp /tmp/" . $pfile . ".webm video.fosdem.org:$uploadpath   true";
        say "mv /tmp/" . $pfile . ".webm .";
 
That script uses avconv to read one or more .dv files and transcode them into a single .dv file with all the start- or end-junk removed. It uses /tmp rather than the working directory, since the working directory was somewhere on the network, and if you're going to write several gigabytes of data to an intermediate file, it's usually a good idea to write them to a local filesystem rather than to a networked one. Pretty boring. It finally calls dv2webm on the resulting .dv file. That script looks like this:
#!/bin/bash
set -e
newfile=$(basename $1 .dv).webm
wavfile=$(basename $1 .dv).wav
wavfile=$(readlink -f $wavfile)
normalfile=$(basename $1 .dv)-normal.wav
normalfile=$(readlink -f $normalfile)
oldfile=$(readlink -f $1)
echo -e "\033]0;Pass 1: $newfile\007"
gst-launch-0.10 webmmux name=mux ! fakesink \
  uridecodebin uri=file://$oldfile name=demux \
  demux. ! ffmpegcolorspace ! deinterlace ! vp8enc multipass-cache-file=/tmp/vp8-multipass multipass-mode=1 threads=2 ! queue ! mux.video_0 \
  demux. ! progressreport ! audioconvert ! audiorate ! tee name=t ! queue ! vorbisenc ! queue ! mux.audio_0 \
  t. ! queue ! wavenc ! filesink location=$wavfile
echo -e "\033]0;Audio normalize: $newfile\007"
sox --norm $wavfile $normalfile
echo -e "\033]0;Pass 2: $newfile\007"
gst-launch-0.10 webmmux name=mux ! filesink location=$newfile \
  uridecodebin uri=file://$oldfile name=video \
  uridecodebin uri=file://$normalfile name=audio \
  video. ! ffmpegcolorspace ! deinterlace ! vp8enc multipass-cache-file=/tmp/vp8-multipass multipass-mode=2 threads=2 ! queue ! mux.video_0 \
  audio. ! progressreport ! audioconvert ! audiorate ! vorbisenc ! queue ! mux.audio_0
rm $wavfile $normalfile
... and is a bit more involved. Multi-pass encoding of video means that we ask the encoder to first encode the file but store some statistics into a temporary file (/tmp/vp8-multipass, in our script), which the second pass can then reuse to optimize the transcoding. Since DV uses different ways of encoding things than does VP8, we also need to do a color space conversion (ffmpegcolorspace) and deinterlacing (deinterlace), but beyond that the video line in the first gstreamer pipeline isn't very complicated. Since we're going over the file anyway and we need the audio data for sox, we add a tee plugin at an appropriate place in the audio line in the first gstreamer pipeline, so that we can later on pick up that same audio data an write it to a wav file containing linear PCM data. Beyond the tee, we go on and do a vorbis encoding, as is needed for the WebM format. This is not actually required for a first pass, but ah well. There's some more conversion plugins in the pipeline (specifically, audioconvert and audiorate), but those are not very important. We next run sox --norm on the .wav file, which does a fully automated audio normalisation on the input. Audio normalisation is the process of adjusting volume levels so that the audio is not too loud, but also not too quiet. Sox has pretty good support for this; the default settings of its --norm parameter make it adjust the volume levels so that the highest peak will just about reach the highest value that the output format can express. As such, you have no clipping anywhere in the file, but also have an audio level that is actually useful. Next, we run a second-pass encoding on the input file. This second pass uses the statistics gathered in the first pass to decide where to put its I- and P-frames so that they are placed at the most optimal position. In addition, rather than reading the audio from the original file, we now read the audio from the .wav file containing the normalized audio which we produced with sox, ensuring the audio can be understood. Finally, we remove the intermediate audio files we created; and the shell script which was generated by perl also contained an rm command for the intermediate .dv file. Some of this is pretty horrid, and I never managed to clean it up enough so it would be pretty (and now is not really the time). However, it Just Works(tm), and I am happy to report that it continues to work with gstreamer 1.0, provided you replace the ffmpegcolorspace by an equally simple videoconvert, which performs what ffmpegcolorspace used to perform in gstreamer 0.10.

8 August 2015

Wouter Verhelst: Backing up with tar

The tape archiver, better known as tar, is one of the older backup programs in existence. It's not very good at automated incremental backups (for which bacula is a good choice), but it can be useful for "let's take a quick snapshot of the current system" type of situations. As I'm preparing to head off to debconf tomorrow, I'm taking a backup of my n-1 laptop (which still contains some data that I don't want to lose) so it can be reinstalled and used by the Debconf video team. While I could use a "proper" backup system, running tar to a large hard disk is much easier. By default, however, tar won't preserve everything, so it is usually a good idea to add some extra options. This is what I' mrunning currently:
sudo tar cvpaSf player.local:carillon.tgz --rmt-command=/usr/sbin/rmt --one-file-system /
which breaks down to create tar archive, verbose output, preserve permissions, automatically determine compression based on file extension, handle Sparse files efficiently, write to a file on a remote host using /usr/sbin/rmt as the rmt program, don't descend into a separate filesystem (since I don't want /proc and /sys etc to be backed up), and back up my root partition. Since I don't believe there's any value to separate file systems on a laptop, this will back up the entire contents of my n-1 laptop to the carillon.tgz in my home directory on player.local.

24 May 2015

Wouter Verhelst: Fixing CVE-2015-0847 in Debian

Because of CVE-2015-0847 and CVE-2013-7441, two security issues in nbd-server, I've had to updates for nbd, for which there are various supported versions: upstream, unstable, stable, oldstable, oldoldstable, and oldoldstable-backports. I've just finished uploading security fixes for the various supported versions of nbd-server in Debian. There're various relevant archives, and unfortunately it looks like they all have their own way of doing things regarding security: While I understand how the differences between the various approaches have come to exist, I'm not sure I understand why they are necessary. Clearly, there's some room for improvement here. As anyone who reads the above may see, doing an upload for squeeze-lts is in fact the easiest of the three "stable" approaches, since no intermediate steps are required. While I'm not about to advocate dropping all procedures everywhere, a streamlining of them might be appropriate.

19 April 2015

Wouter Verhelst: Youn Sun Nah 5tet: Light For The People

About a decade ago, I played in the (now defunct) "Jozef Pauly ensemble", a flute choir connected to the musical academy where I was taught to play the flute. At the time, this ensemble had the habit of goin on summer trips every year; sometimes these trips were large international concert tours (like our 2001 trip to Australia), but that wasn't always the case; there have also been smaller trips, like the 2002 one to the French Ardennes. While there, we went on a day trip to the city of Reims. As a city close to the front in the first world war, it has a museum dedicated to that subject that I remembered going to. But the fondest memory of that day was going to a park where a podium was set up, with a few stacks of fold-up chairs standing nearby. I took one and listened to the music. That was the day when I realized that I kindof like jazz. I had come into contact with Jazz before, but it had always been something to be used as a kind of musical wallpaper; something you put on, but don't consciously listen to. Watching this woman sing, however, was a different kind of experience altogether. I'm still very fond of her rendition of "Besame Mucho". After having listened to the concert for about two hours, they called it quits, but did tell us that there was a record which you could buy. Of course, after having enjoyed the afternoon so much, I couldn't imagine not buying it, so that happened. Fast forward several years, in the move from my apartment above my then-office to my current apartment (just around the corner), the record got put into the wrong box, and when I unpacked things again it got lost; permanently, I thought. Since I also hadn't digitized it yet at the time, I haven't listened to it anymore in quite a while. But that time came to an end today. The record which I thought I'd lost wasn't, it was just in a weird place, and while cleaning yesterday, I found it sitting among a bunch of old stuff that I was going to throw out. Putting on the record today made me realize again how good it really is, and I thought that I might want to see if she was still active, and if she might perhaps have made another album. It was great to find out that not only had she made six more albums since the one I bought, she'd also become a lot more known in the Jazz world (which I must admit I don't really follow all that well), and won a number of awards. At the time, Youn Sun Nah was just a (fairly) recent graduate from a particular Jazz school in Paris. Today, she appears to be so much more...

12 April 2015

Wouter Verhelst: LOADays 2015 talk done

I just uploaded my LOADays 2015 slides to slideshare. The talk seems to have been well received; I got a number of positive comments from some attendees, which is always nice. As an aside, during the talk I did a short demo of how to sign something from within Libreoffice using my eID card. Since the slides were made in Libreoffice Impress, the easiest thing to do was just to sign the slides themselves, which worked perfectly well. So, having uploaded, downloaded, and verified these slides, I can now say with 100% certainty that slideshare does not tamper with files you upload. They may reformat them so it's easier to view on a website, but if you click on the download link, you get the original, untampered version. At least that's the case if you sign documents, of course; it's always possible that they check for that and special-case such things. Would surprise me, though.

7 April 2015

Wouter Verhelst: C11 function overloading

About four years ago, the ISO 9899:2011 "C11" standard was announced. At the time, I had a short look at (a draft version of) the standards document, and found a few interesting bits in there. Of course, however, due to it only very recently having been released, I did not have much hope of it being implemented to any reasonable amount anywhere yet. Which turned out to be the case. Even if that wasn't true, writing code that uses C11 features and expecting it to work just about anywhere else would have been a bad idea back then. We're several years down the line now, however, and now the standard has been implemented to a reasonable extent in most compilers. GCC claims its "support [for C11] is at a similar level of completeness to (...) C99 support" since GCC 4.9. Since my laptop has GCC 4.9, I looked at one feature in C11 that I have been wanting to use for a while: Generic selection.
#include <stdint.h>
#include <inttypes.h>
#include <stdio.h>
void say32(uint32_t i)  
    printf("32-bit variable: %" PRId32 "\n", i);
 
void say64(uint64_t i)  
    printf("64-bit variable: %" PRId64 "\n", i);
 
void sayother(int i)  
    printf("This is something else.\n");
 
#define say(X) _Generic((X), uint32_t: say32, uint64_t: say64, default: sayother)(X)
int main(void)  
    uint32_t v32 = 32;
    uint64_t v64 = 64;
    uint8_t v8 = 8;
    say(v32);
    say(v64);
    say(v8);
 
Output of the above:
32-bit variable: 32
64-bit variable: 64
This is something else.
or, "precompiler-assisted function overloading for C". Should be useful for things like:
#define ntoh(X) _Generic((X), int16_t: ntohs, uint16_t: ntohs, int32_t: ntohl, uint32_t: ntohl)(X)
#define hton(X) _Generic((X), int16_t: ntohs, uint16_t: htons, int32_t: ntohl, uint32_t: htonl)(X)
... and if one adds the ntohll found here, it can do 64 bit as well.

Wouter Verhelst: C11 operator overloading

C11 function overloading About four years ago, the ISO 9899:2011 "C11" standard was announced. At the time, I had a short look at (a draft version of) the standards document, and found a few interesting bits in there. Of course, however, due to it only very recently having been released, I did not have much hope of it being implemented to any reasonable amount anywhere yet. Which turned out to be the case. Even if that wasn't true, writing code that uses C11 features and expecting it to work just about anywhere else would have been a bad idea back then. We're several years down the line now, however, and now the standard has been implemented to a reasonable extent in most compilers. GCC claims its "support [for C11] is at a similar level of completeness to (...) C99 support" since GCC 4.9. Since my laptop has GCC 4.9, I looked at one feature in C11 that I have been wanting to use for a while: Generic selection.
#include <stdint.h>
#include <inttypes.h>
#include <stdio.h>
void say32(uint32_t i)  
    printf("32-bit variable: %" PRId32 "\n", i);
 
void say64(uint64_t i)  
    printf("64-bit variable: %" PRId64 "\n", i);
 
void sayother(int i)  
    printf("This is something else.\n");
 
#define say(X) _Generic((X), uint32_t: say32, uint64_t: say64, default: sayother)(X)
int main(void)  
    uint32_t v32 = 32;
    uint64_t v64 = 64;
    uint8_t v8 = 8;
    say(v32);
    say(v64);
    say(v8);
 
Output of the above:
32-bit variable: 32
64-bit variable: 64
This is something else.
or, "precompiler-assisted function overloading for C". Should be useful for things like:
#define ntoh(X) _Generic((X), int16_t: ntohs, uint16_t: ntohs, int32_t: ntohl, uint32_t: ntohl)(X)
#define hton(X) _Generic((X), int16_t: ntohs, uint16_t: htons, int32_t: ntohl, uint32_t: htonl)(X)
... and if one adds the ntohll found here, it can do 64 bit as well.

13 March 2015

Wouter Verhelst: New toy: Fujitsu Lifebook e734

My Lenovo x220, which I've owned for almost four years now (I remember fetching it from the supplier shortly before driving off to Banja Luka), was getting somewhat worn out. The keyboard and the screen had both been replaced at some point already, and the wwan interface had given up as well. The case was all cracked, and the NIC connector wasn't doing very well anymore either; there have been a few cases of me trying to configure the wireless network at a customer, but this being harder than it needs to be because the NIC will only work if I put in the network cable just so, and someone dropped a piece of paper onto the cable. In other words, it was time for a new one. At first I wanted to buy a Lenovo x250, but then I noticed that the Fujitsu came with an i7 4712MQ, which I liked (as today it is still quite exceptional for an ultrabook to have a quadcore processor). Fujitsu also claims up to 9 hours of battery life, but it's not clear to me whether this is supposed to be the case with the default battery only. They also have a battery for the modular bay, which I bought as well (to replace the optical drive whic I sometimes use, but only rarely), and on top of that it came with a free port replicator. Not all is well, however. In the x220, getting the WWAN interface to work involved some creative use of chat against /dev/ttyACM0 wherein I issue a few AT commands to put the WWAN interface into a particular mode, and from then on the WWAN interface is just a regular Ethernet interface on which I can do DHCP. The new laptop has a "Sierra Wireless, Inc." WWAN interface (USB id 1199:9041) which annoyingly doesn't seem to expose the ttyACM (or similar) devices, and I'm not sure what to use instead. Just trying to do DHCP doesn't work -- yes, I tried. Unfortunately, the keyboard isn't very good; it's of the bubble gum type, and I keep getting annoyed at it not picking up my keystrokes all the time. When I'm at home or at my main customer, I have a Das Keyboard Ultimate S (3rd (customer) and 4th (home) generation), so it's only a problem when I'm not at home, but it's still extremely annoying. There is a "backlight" function in that keyboard, but that's not something I think I'll ever use (hint: "das keyboard ultimate s"). The display can't do more than 1366x768, which is completely and utterly wrong for a computer -- but it's the same thing as my x220, so it's not really a regression. The "brightness" ACPI keys don't seem to work. I may have to fiddle with some ACPI settings at some point, I suppose, but it's not a major problem. When I plugged it in, I noticed that fdpowermon ignored the second battery. I had originally written fdpowermon with support for such a second battery, but as my x220 had only one, I never tested it. Apparently there was a bug, but that's been fixed now -- at least in unstable. On the good side of the equation, it has three USB3 ports in the laptop, and four in the port replicator, with no USB2; this is a major leap forwards from the one USB3 and six USB2 in the x220. A positive surprise was the CCID smartcard reader that I somehow missed while reading the specs, but which -- given my current major customer, is very welcome, indeed. Update: After having used it a few days, there were a few minor annoyances:

2 March 2015

Wouter Verhelst: NBD 3.9

I just released NBD 3.9 When generating the changelog, I noticed that 3.8 happened two weeks shy of a year ago, which is far too long. As a result, the new release has many new features: Get it at the usual place.

26 February 2015

Wouter Verhelst: Dear non-Belgian web developer,

Localization in the web context is hard, I know. To make things easier, it may seem like a good idea to use GeoIP to detect what country an IP is coming from and default your localization based on that. While I disagree with that premise, this blog post isn't about that. Instead, it's about the fact that most of you get something wrong about this little country. I know, I know. If you're not from here, it's difficult to understand. But please get this through your head: Belgium is not a French-speaking country. That is, not entirely. Yes, there is a large group of French-speaking people who live here. Mostly in the south. But if you check the numbers, you'll find that there are, in fact, more people in Belgium who speak Dutch rather than French. Not by a very wide margin, mind you, but still by a wide enough margin to be significant. Wikipedia claims the split is 59%/41% Dutch/French; I don't know how accurate those numbers are, but they don't seem too wrong. So please, pretty please, with sugar on top: next time you're going to do a localized website, don't assume my French is better than my English. And if you (incorrectly) do, then at the very least make it painfully obvious to me where the "switch the interface to a different language" option in your website is. Because while it's annoying to be greeted in a language that I'm not very good at, it's even more annoying to not be able to find out how to get the correctly-localized version. Thanks.

20 February 2015

Wouter Verhelst: LOADays 2015

Looks like I'll be speaking at LOADays again. This time around, at the suggestion of one of the organisers, I'll be speaking about the Belgian electronic ID card, for which I'm currently employed as a contractor to help maintain the end-user software. While this hasn't been officially confirmed yet, I've been hearing some positive signals from some of the organisers. So, under the assumption that my talk will be accepted, I've started working on my slides. The intent is to explain how the eID middleware works (in general terms), how the Linux support is supposed to work, and what to do when things fail. If my talk doesn't get rejected at the final hour, I will continue my uninterrupted "speaker at loadays" streak, which has started since loadays' first edition...

14 February 2015

Wouter Verhelst: Docker

... is the new hype these days. Everyone seems to want to be part of it; even Microsoft wants to allow Docker to run on its platform. How they visualise that is slightly beyond me, seen as how Docker is mostly a case of "run a bunch of LXC instances", which by their definition can't happen on Windows. Presumably they'll just run a lot more VMs, then, which is a possible workaround. Or maybe Docker for Windows will be the same in concept, but not in implementation. I guess the future will tell. As I understand the premise, the idea of Docker is that getting software to run on "all" distributions is a Hard Problem[TM], so in a Docker thing you just define that this particular stuff is meant to run on top of this and this and that environment, and Docker then compartmentalises everything for you. It should make things easier to maintain, and that's a good thing. I'm not a fan. If the problem that Docker tries to fix is "making software run on all platforms is hard", then Docker's "solution" is "I give up, it's not possible". That's sad. Sure, having a platform which manages your virtualisation for you, without having to manually create virtual machines (or having to write software to do so) is great. And sure, compartmentalising software so that every application runs in its own space can help towards security, manageability, and a whole bunch of other advantages. But having an environment which says "if you want to run this applicaiton, I'll set up a chroot with distribution X for you; if you want to run this other application, I'll set up a chroot with distribution Y for you; and if you want to run yet this other application yere, I'll start doing a chroot with distribution Z for you" will, in the end, get you a situation where, if there's another bug in libc6 or libssl, you now have a nightmare trying to track down all the different versions in all the docker instances to make sure they're all fixed. And while it may work perfectly well on the open Internet, if you're on a corporate network with a paranoid firewall and proxy, downloading packages from public mirrors is harder than just creating a local mirror instead. Which you now have to do not only for your local distribution of choice, but also for the distributions of choice of all the developers of the software you're trying to use. Which may result in more work than just trying to massage the software in question to actually bloody well work, dammit. I'm sure Docker has a solution for some or all of the problems it introduces, and I'm not saying it doesn't work in practice. I'm sure it does fix some part of the "Making software run on all platforms is hard" problem, and so I might even end up using it at some point. But from an aesthetical point of view, I don't think Docker is a good system. I'm not very fond of giving up.

8 January 2015

Wouter Verhelst: ExtreMon example

About a month ago, I blogged about extremon. As a reminder, ExtreMon is a monitoring tool that allows you to view things as they are happening, rather than with the ~5 minute delay that munin gives you, and also avoiding the quad-state limitation of Nagios' "good", "bad", "ugly", and "unknown" states. No, they're not really called that. Yes, I know you knew that. Anyway. In my blog post, I explained how you can set up ExtreMon, and I also set up a fairly limited demo version on my own server. But I have since realized that while it is functional, it doesn't actually show why ExtreMon is so great. In an effort to remedy that, I present you an example of what ExtreMon can do. Let's start with a screenshot of the ExtreMon console at the customer for which I spent time trying to figure out how to get it up and running: Click for full sized version. You'll note that even in that full-sized version, many things are unreadable. This is because the ExtreMon console allows one to move around (right mouse button drag for zoom; left mouse button drag for moving around; control+RMB for rotate; center mouse button to reset to default); so what matters is that everything fits on the screen, not whether it is all readable (if you need to read, you zoom). The image shows 18 rectangles. Each rectangle represents a single machine in this particular customer's HPC cluster. The top three rectangles are the cluster's file servers; the rest are its high performance nodes. You'll note that the left fileserver has 8 processor cores (top row), 8 network cards (bottom row, left part), and it also shows information on its memory usage (bottom row, small rectangle in the middle) as well as its NFS client and server procedure calls (bottom row, slightly larger rectangles to the right). This file server is the one on which I installed ZFS a while back; hence the large amount of disks visible in the middle row. The leftmost disk is the root filesystem (which is an ext4 off a hardware RAID1); the two rightmost "disks" are the PCIe-attached SSDs which are used for the ZFS L2ARC and write log. The other disks in this file server nicely show how ZFS does write load balancing over all its disks. The second file server has a hardware RAID1 on which it stores all its data; as such, there is only one disk graph there. It is also somewhat more limited in network, as it has only two NICs. It does, however, also have 8 cores. The last file server has no more than four processor cores; in addition, it also does not have a hardware RAID controller, so it must use software RAID over its four hard disks. This server is used for archival purposes, mostly, since it is insufficient for most anything else. As said, the other nodes are the "compute nodes", where the hard work is done. Most of these compute nodes have 16 cores each; two have 12 instead. When this particular screenshot was taken, four of the nodes (the ones showing red in their processor graphs) were hard at work; the others seem to have been mostly idling. In addition to the familiar memory, NFS (client only), network, and processor graphs, these nodes also show a "swap space" graph (just below the memory one), which seems fine for most nodes, except for the bottom left one (which shows a few bars that are coloured yellow rather than green). The green/yellow/red stuff is supposed to represent the "ok", "warning", "bad" states that would be familiar from Nagios. In this particular case, however, where "processor is busy all the time" is actually a wanted state, a low amount of idleness on the part of the processor isn't actually a problem, on the contrary. I did consider, therefore, to modify the ExtreMon configuration so that the processor graphs would not show red when the system was under high load; however, I found that differences in colour like this actually makes it more easy to see, at a glance, which machines are busy -- and that's one of the main reasons why we wanted to set this up. If you look carefully, you can find a particular processor core in the graph which shows 100% usage for "idle", "system", and "softirq", at the same time. Obviously that can't be the case, so there's a bug somewhere. Frank seems to believe it is a bug in CollectD; I haven't verified that. At any rate, though, this isn't usually a problem, due to the high update frequency of ExtreMon. The amount of data that's flowing through ExtreMon is amazing: Which renders a grand total of 2887 data points that are shown in this particular screenshot; and then I'm not even counting all the intermediate values, some of which also pass through ExtreMon. Nor am I counting the extra bits which have since been added (this screenshot is a few days old, now, and I'm still finetuning things). Yet even so, ExtreMon manages to update those values once every few seconds, in the worst case. As a result, the display isn't static for a moment, constantly moving and updating data so that what you see is never out of date for more than a second or two. Awesome.

31 December 2014

Wouter Verhelst: Perl 'issues'

I just watched a CCC talk in which the speaker claims Perl is horribly broken. Watching it was fairly annoying however, since I had to restrain myself from throwing things at the screen. If you're going to complain about the language, better make sure you actually understand the language first. I won't deny that there are a few weird constructions in there, but hey. The talk boils down to a claim that perl is horrible, because the list "data type" is "broken". First of all, Netanel, in Perl, lists are not arrays. Yes, that's confusing if you haven't done more than a few hours of Perl, but hear me out. In Perl, a list is an enumeration of values. A variable with an '@' sigil is an array; a construct consisting of an opening bracket ('(') followed by a number of comma- or arrow-separated values (',' or '=>'), followed by a closing bracket, is a list. Whenever you assign more than one value to an array or a hash, you need to use a list to enumerate the values. Subroutines in perl also use lists as arguments or return values. Yes, that last bit may have been a mistake. Perl has a concept of "scalar context" and "list context". A scalar context is what a sub is in when you assign the return value of your sub to a scalar; a list context is when you assign the return value of your sub to an array or a hash, or when you use the list construct (the thing with brackets and commas) with sub calls (instead of hardcoded values or variables) as the individual values. This works as follows:
sub magic  
    if (wantarray())  
        print "You're asking for a list!";
        return ('a', 'b', 'c');
      else  
        print "You're asking for a scalar!";
        return 'a';
     
 
print ("list: ", magic(), "\n");
print "scalar: " . magic() . "\n";
The above example will produce the following output:
You're asking for a list!
list: abc
You're asking for a scalar!
scalar: a
What happens here? The first print line creates a list (because things are separated by commas); the second one does not (the '.' is perl's string concatenation operator; as you can only concatenate scalars, the result is that you call the magic() sub in scalar context). Yes, seen as how arrays are not lists, the name of the wantarray() sub is horribly chosen. Anyway. It is documented that lists cannot be nested. Lists can only be one-dimensional constructs. If you create a list, and add another list as an element (or something that can be converted to a list, like an array or a hash), then the result is that you get a flattened list. If you don't want a flattened list, you need to use a reference instead. A reference is a scalar value that, very much like a pointer in C, contains a reference to another variable. This other variable can be an array, a hash, or a scalar. But it cannot be a list, because it must be a variable -- and lists cannot be variables. If you need to create multi-dimensional constructs, you need to use references. Taking a reference is done by prepending a backslash to whatever it is you're trying to take a reference of; or, in the case of arrays of hashes, one can create an anonymous array or hash with [] resp . E.g., if you want to add a non-flattened array to a list, you instead create a reference to an array, like so:
$arrayref = [ 'this', 'is', 'an', 'anonymous', 'array'];
you can now create a multi-dimensional construct:
@multiarray = ('elem1', $arrayref);
Or you can do that in one go:
@multiarray = ('elem1', [ 'this', 'is', 'an', 'anonymous', 'array']);
Alternatively, you can create a non-anonymous array first:
@onedimarray = ('this', 'is', 'not', 'an', 'anonymous', 'array');
@multiarray = ('elem1', \@onedimarray);
In perl, curly brackets can be used to create a reference to anonymous hashes, whereas square brackets can be used to create a reference to anonymous arrays. This is all a basic part of the language; if you don't understand that, you simply don't understand Perl. In other words, whenever you see someone doing this:
%hash =  'a' => 'b' ;
or
@array = [ '1', '2' ];
you can say that they don't understand the language. For reference, the assignment to %hash will result in an (unusable) hash with a single key that is a reference to an anonymous hash (which cannot be accessed anymore) and a value of undef; the assignment to @array will result in a two-dimensional array with one element in the first dimension, and two elements in the second. The CGI.pm fix which Natanel dismisses in the Q&A part of the talk as a "warning" which won't help (because it would be too late) is actually a proper fix, which should warn people in all cases. That is, if you do this:
%hash =   'name' => $name, 'password' => $cgi->param('password')  ;
then CGI.pm's param() sub will notice that it's being called in list context, and issue a warning -- regardless of whether the user is passing one or two password query-parameters. It uses the wantarray() sub, and produces a warning if that returns true. In short, Perl is not the horribly broken construct that Natanel claims it to be. Yes, there are a few surprises (most of which exist for historical reasons), and yes, those should be fixed. This is why the Perl community has redone much of perl for Perl 6. But the fact that there are a few surprises doesn't mean the whole language is broken. There are surprises in most languages; that is a fact of life. Yes, the difference between arrays and hashes on the one hand, and lists on the other hand, is fairly confusing; it took me a while to understand this. But once you get the hang of it, it's not all that difficult. And then these two issues that Natanel found (which I suppose could be described as bugs in the core modules) aren't all that surprising anymore. So, in short: What I do agree with is that if you want to use a language, you should understand its features. Unfortunately, this single line in the final slide of Natanel's talk is just about the only thing in the whole talk that sortof made sense to me. Ah well.

19 December 2014

Wouter Verhelst: joytest UI improvements

After yesterday's late night accomplishments, today I fixed up the UI of joytest a bit. It's still not quite what I think it should look like, but at least it's actually usable with a 27-axis, 19-button "joystick" (read: a PS3 controller). Things may disappear off the edge of the window, but you can scroll towards it. Also, I removed the names of the buttons and axes from the window, and installed them as tooltips instead. Few people will be interested in the factoid that "button 1" is a "BaseBtn4", anyway. The result now looks like this: If you plug in a new joystick, or remove one from the system, then as soon as udev finishes up creating the necessary device node, joytest will show the joystick (by name) in the treeview to the left. Clicking on a joystick will show that joystick's data to the right. When one pushes a button, the relevant checkbox will be selected; and when one moves an axis, the numbers will start changing. I really should have some widget to actually show the axis position, rather than some boring numbers. Not sure how to do that.

Wouter Verhelst: joytest.png

18 December 2014

Wouter Verhelst: Introducing libjoy

I've owned a Logitech Wingman Gamepad Extreme since pretty much forever, and although it's been battered over the years, it's still mostly functional. As a gamepad, it has 10 buttons. What's special about it, though, is that the device also has a mode in which a gravity sensor kicks in and produces two extra axes, allowing me to pretend I'm really talking to a joystick. It looks a bit weird though, since you end up playing your games by wobbling the gamepad around a bit. About 10 years ago, I first learned how to write GObjects by writing a GObject-based joystick API. Unfortunately, I lost the code at some point due to an overzealous rm -rf call. I had planned to rewrite it, but that never really happened. About a year back, I needed to write a user interface for a customer where a joystick would be a major part of the interaction. The code there was written in Qt, so I write an event-based joystick API in Qt. As it happened, I also noticed that jstest would output names for the actual buttons and axes; I had never noticed this, because due to my 10 buttons and 4 axes, which by default produce a lot of output, the jstest program would just scroll the names off my screen whenever I plugged it in. But the names are there, and it's not too difficult. Refreshing my memory on the joystick API made me remember how much fun it is, and I wrote the beginnings of what I (at the time) called "libgjs", for "Gobject JoyStick". I didn't really finish it though, until today. I did notice in the mean time that someone else released GObject bindings for javascript and also called that gjs, so in the interest of avoiding confusion I decided to rename my library to libjoy. Not only will this allow me all kinds of interesting puns like "today I am releasing more joy", it also makes for a more compact API (compare joy_stick_open() against gjs_joystick_open()). The library also comes with a libjoy-gtk that creates a GtkListStore* which is automatically updated as joysticks are added and removed to the system; and a joytest program, a graphical joystick test program which also serves as an example of how to use the API. still TODO: What's there is functional, though. Update: if you're going to talk about code, it's usually a good idea to link to said code. Thanks, Emanuele, for pointing that out ;-)

Next.

Previous.